Information processing apparatus, information processing method, and storage medium

ABSTRACT

An information processing apparatus includes: an acquisition unit that acquires a two-dimensional image of a person, and a registered image group including two-dimensional registered images and three-dimensional registered images of a plurality of registrants; and a selection unit that selects a type of a registered image to be used for matching from one of the two-dimensional registered images and the three-dimensional registered images based on the two-dimensional image, prior to the matching the two-dimensional image with the registered image group.

This application is a National Stage Entry of PCT/JP2019/050707 filed onDec. 24, 2019, which claims priority from Japanese Patent Application2018-244593 filed on Dec. 27, 2018, the contents of all of which areincorporated herein by reference, in their entirety.

TECHNICAL FIELD

This disclosure relates to an information processing apparatus, aninformation processing method, and a storage medium.

BACKGROUND ART

In recent years, an apparatus that identifies a person by matching animage of the person captured by a security camera or the like with animage of a registrant registered in a database in advance is widely useddue to development of image recognition techniques. In the nature ofoperation, however, there is a case where a user has to visually check amatching result.

Patent Literature 1 discloses a 2D/3D composite matching apparatus whichgenerates a 3D matching face image from a stereo image obtained byphotographing an object of matching from two directions and switches amatching mode to a 2D matching mode or a 3D matching mode based onwhether or not the 3D matching face image faces the front.

CITATION LIST Patent Literature

-   PTL 1: Japanese Patent Application Laid-Open No. 2011-203791

SUMMARY OF INVENTION Technical Problem

The apparatus disclosed in Patent Literature 1 improves the accuracy ofmatching process by selecting an appropriate matching mode based on theorientation of a 3D matching face image. However, since the apparatus isconfigured to generate a 3D matching face image from the stereo image ofthe matching object to be used for determining the matching mode, ittakes time from the time when the matching object is photographed untilthe matching processing is completed.

Accordingly, this disclosure has been made in view of the above problemand intends to provide an information processing apparatus, aninformation processing method, and a storage medium that can increasethe speed of the matching process for person images.

Solution to Problem

According to one example aspect of this disclosure, provided is aninformation processing apparatus including: an acquisition unit thatacquires a two-dimensional image of a person, and a registered imagegroup including two-dimensional registered images and three-dimensionalregistered images of a plurality of registrants; and a selection unitthat selects a type of a registered image to be used for matching fromone of the two-dimensional registered images and the three-dimensionalregistered images based on the two-dimensional image, prior to thematching the two-dimensional image with the registered image group.

According to another example aspect of this disclosure, provided is aninformation processing method including: acquiring a two-dimensionalimage of a person, and a registered image group includingtwo-dimensional registered images and three-dimensional registeredimages of a plurality of registrants; and selecting a type of aregistered image to be used for matching from one of the two-dimensionalregistered images and the three-dimensional registered images based onthe two-dimensional image, prior to the matching the two-dimensionalimage with the registered image group.

According to yet another example aspect of this disclosure, provided isa storage medium storing a program that causes a computer to perform:acquiring a two-dimensional image of a person, and a registered imagegroup including two-dimensional registered images and three-dimensionalregistered images of a plurality of registrants; and selecting a type ofa registered image to be used for matching from one of thetwo-dimensional registered images and the three-dimensional registeredimages based on the two-dimensional image, prior to the matching thetwo-dimensional image with the registered image group.

According to this disclosure, an information processing apparatus, aninformation processing method, and a program that can increase the speedof the matching process for person images.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating the entire configuration example of aninformation matching system in a first example embodiment.

FIG. 2 is a block diagram illustrating the function of an image matchingapparatus in the first example embodiment.

FIG. 3 is a diagram illustrating one example of registrant informationstored in a storage unit in the first example embodiment.

FIG. 4A is a diagram illustrating a positional relationship between faceorientation of a matching target and an image capturing apparatus in thefirst example embodiment.

FIG. 4B is a diagram illustrating a positional relationship between faceorientation of a matching target and an image capturing apparatus in thefirst example embodiment.

FIG. 5 is a block diagram illustrating a hardware configuration exampleof the image matching apparatus in the first example embodiment.

FIG. 6 is a flowchart illustrating one example of a matching process inthe first example embodiment.

FIG. 7 is a diagram illustrating one example of a candidate displaywindow in the first example embodiment.

FIG. 8 is a flowchart illustrating one example of a generation processof a composite image in the first example embodiment.

FIG. 9 is a diagram illustrating one example of a person check window inthe first example embodiment.

FIG. 10 is a flowchart illustrating one example of a generation processof a composite image in a second example embodiment.

FIG. 11A is a diagram illustrating a two-dimensional image of a matchingtarget in the second example embodiment.

FIG. 11B is a diagram illustrating a three-dimensional registered imageof a candidate in the second example embodiment.

FIG. 11C is a diagram illustrating a horizontally composed image of thetwo-dimensional image and the three-dimensional registered image in thesecond example embodiment.

FIG. 11D is a diagram illustrating a vertically composed image of thetwo-dimensional image and the three-dimensional registered image in thesecond example embodiment.

FIG. 12A is a diagram illustrating a two-dimensional image of a matchingtarget in the second example embodiment.

FIG. 12B is a diagram illustrating a three-dimensional edited image of acandidate in the second example embodiment.

FIG. 12C is a diagram illustrating a horizontally composed image of thetwo-dimensional image and the three-dimensional edited image in thesecond example embodiment.

FIG. 12D is a diagram illustrating a three-dimensional edited image inthe second example embodiment.

FIG. 13A is a diagram illustrating a two-dimensional image of a matchingtarget in the second example embodiment.

FIG. 13B is a diagram illustrating a three-dimensional registered imageof a candidate in the second example embodiment.

FIG. 13C is a diagram illustrating a horizontally composed image of thetwo-dimensional image and the three-dimensional registered image in thesecond example embodiment.

FIG. 13D is a diagram illustrating a vertically composed image of thetwo-dimensional image and the three-dimensional registered image in thesecond example embodiment.

FIG. 14A is a diagram illustrating a two-dimensional image of a matchingtarget in the second example embodiment.

FIG. 14B is a diagram illustrating a three-dimensional edited image of acandidate in the second example embodiment.

FIG. 14C is a diagram illustrating a horizontally composed image of thetwo-dimensional image and the three-dimensional edited image in thesecond example embodiment.

FIG. 14D is a diagram illustrating a three-dimensional edited image inthe second example embodiment.

FIG. 15A is a diagram illustrating a two-dimensional image of a matchingtarget in the second example embodiment.

FIG. 15B is a diagram illustrating a three-dimensional registered imageof a candidate in the second example embodiment.

FIG. 15C is a diagram illustrating a horizontally composed image of thetwo-dimensional image and the three-dimensional registered image in thesecond example embodiment.

FIG. 15D is a diagram illustrating a vertically composed image of thetwo-dimensional image and the three-dimensional registered image in thesecond example embodiment.

FIG. 16A is a diagram illustrating a two-dimensional image of a matchingtarget in the second example embodiment.

FIG. 16B is a diagram illustrating a three-dimensional edited image of acandidate in the second example embodiment.

FIG. 16C is a diagram illustrating a horizontally composed image of thetwo-dimensional image and the three-dimensional edited image in thesecond example embodiment.

FIG. 16D is a diagram illustrating a three-dimensional edited image inthe second example embodiment.

FIG. 17A is a diagram illustrating a two-dimensional image of a matchingtarget in the second example embodiment.

FIG. 17B is a diagram illustrating a three-dimensional registered imageof a candidate in the second example embodiment.

FIG. 17C is a diagram illustrating a horizontally composed image of thetwo-dimensional image and the three-dimensional registered image in thesecond example embodiment.

FIG. 17D is a diagram illustrating a vertically composed image of thetwo-dimensional image and the three-dimensional registered image in thesecond example embodiment.

FIG. 18A is a diagram illustrating a two-dimensional image of a matchingtarget in the second example embodiment.

FIG. 18B is a diagram illustrating a three-dimensional edited image of acandidate in the second example embodiment.

FIG. 18C is a diagram illustrating a horizontally composed image of thetwo-dimensional image and the three-dimensional edited image in thesecond example embodiment.

FIG. 18D is a diagram illustrating a three-dimensional edited image inthe second example embodiment.

FIG. 19 is a flowchart illustrating one example of a selection processof a matching mode in a third example embodiment.

FIG. 20 is a block diagram illustrating the function of an informationprocessing apparatus in a fourth example embodiment.

DESCRIPTION OF EMBODIMENTS

Illustrative example embodiments of this disclosure will be describedbelow with reference to the drawings. In the drawings, the same elementsor corresponding elements are labeled with the same reference, and thedescription thereof may be omitted or simplified.

First Example Embodiment

First, the configuration of an image matching system 1 in the presentexample embodiment will be described with reference to FIG. 1 to FIG. 5. FIG. 1 is a diagram illustrating the entire configuration example ofthe image matching system 1 in the present example embodiment. The imagematching system 1 is an information processing system that has an imagematching apparatus 10, an image capturing apparatus 20, and a readingapparatus 30. Each apparatus is connected to a network 40 such as aLocal Area Network (LAN), the Internet, or the like.

The image matching apparatus 10 is an information processing apparatusthat, in an event site, a theme park, a traffic facility and a publicinstitution, or the like, for example, matches biometrics informationobtained by an image of a matching target person (hereafter, referred toas “matching target”) with a biometrics information group of a pluralityof registrants that is registered in a database in advance. Thebiometrics information may be a face image, a fingerprint image, an irisimage, a finger vein image, a palm print image, a palm vein image, orthe like.

Note that the term “biometrics information” in the present exampleembodiment means a face image and a feature amount extracted from theface image. Further, a plurality of face images of registrants areobtained for each registrant by capturing a face of a person not onlyfrom the front but also at a plurality of capturing angles whenmembership registration is performed, for example. A feature amountextracted from a face image may be an amount indicating a feature of aface such as a position of a characteristic part such as a pupil, anose, a mouth end, for example.

The image capturing apparatus 20 is a network camera such as a securitycamera installed in the facility to be managed. The image capturingapparatus 20 outputs a capturing image obtained by capturing apredetermined region of a facility to the image matching apparatus 10,for example. Note that, while the image capturing apparatus 20 isconnected to the image matching apparatus 10 via the network 40 by wirein FIG. 1 , the example embodiment is not limited thereto. The imagecapturing apparatus 20 may be connected to the image matching apparatus10 wirelessly. Thus, the image capturing apparatus 20 may be a userterminal such as a smartphone, a tablet terminal, a personal computer,or the like.

The reading apparatus 30 is a scanner apparatus that optically reads aprint such as a photograph or an apparatus that reads data stored in anelectronic storage medium such as an IC card, a USB memory, and a disc.The reading apparatus 30 outputs the read image data to the imagematching apparatus 10.

FIG. 2 is a function block diagram of the image matching apparatus 10 inthe present example embodiment. The image matching apparatus 10 has astorage unit 11, an image acquisition unit 12, an image analysis unit13, a matching mode select unit 14, a matching unit 15, a displayinformation generation unit 16, and a composite image generation unit17.

The storage unit 11 stores a two-dimensional face image andthree-dimensional images of a plurality of registrants as registrantinformation. The registrant information may be a black list, a whitelist, a VIP list, an employee list, or the like. FIG. 3 is a diagramillustrating one example of registrant information stored in a storageunit 11 in the present example embodiment. Herein, an example of a dataitem of the registrant information may be a registrant ID, a name, anaddress, a two-dimensional image (2D face image), and athree-dimensional image (3D face image). The data item of the registrantinformation is not limited to the item illustrated in FIG. 3 and mayfurther include attribute information of a registrant such as a facefeature amount extracted from a face image, the age, the body height, abody type (slender type/normal type/obesity type), a birthplace, a bodycharacteristic such as the presence or absence of a mole or a scar, acareer, and an academic background of the registrant. Furthermore, thedata item of the registrant information may further include attributeinformation of a file, such as an image registration date.

Furthermore, in FIG. 3 , while a face image facing the front isillustrated as an example as a two-dimensional image (2D face image), aface orientation of a two-dimensional image stored in the storage unit11 is not limited to only the front. For example, a face orientation maybe a two-dimensional image when viewed from the left or right side or atwo-dimensional image when viewed from an oblique direction (forexample, 45 degrees). Furthermore, a plurality of two-dimensional imagescorresponding to a plurality of face orientations may be stored. Aplurality of two-dimensional images may be created from athree-dimensional image or may be created together in a process ofcreating a three-dimensional image. When the storage unit 11 stores atwo-dimensional image obtained by capturing the front, the right side,or the left side of a face, for example, a face image when facingdiagonally forward, which corresponds to a capturing condition betweenthe front and the right side, may be supplemented with a 3D model(three-dimensional image).

The image acquisition unit 12 acquires a two-dimensional image of amatching target from the image capturing apparatus 20 or the readingapparatus 30. Furthermore, the image acquisition unit 12 acquires atwo-dimensional registered image and three-dimensional registered imagesof a plurality of registrants from the storage unit 11 in response to arequest from the display information generation unit 16 or the compositeimage generation unit 17.

The image analysis unit 13 analyzes a two-dimensional image acquired inthe image acquisition unit 12 and detects the face image correspondingto a face region of a matching target. Furthermore, the image analysisunit 13 detects the face orientation of a matching target based on thedistance and the positional relationship or the like between featurepositions (eyes, a nose, a mouth, or the like) in a face image. The faceorientation may be calculated based on a state where a matching targetfaces the front as a reference, for example.

The matching mode select unit 14 selects a type of a registered imageused for matching from one of a two-dimensional registered image and athree-dimensional registered image based on a two-dimensional image,prior to matching a two-dimensional image with a registered image group.

In the present example embodiment, a matching mode for matching atwo-dimensional image with a two-dimensional registered image isreferred to as “2D matching mode” and a matching mode for matching atwo-dimensional image with a three-dimensional registered image isreferred to as “3D matching mode”. The 2D matching mode has a highprocessing speed and can realize a real-time property. On the contrary,a matching accuracy decreases when a face orientation is not the same asthe orientation in a face photograph listed on the registrant list (forexample, the front). Furthermore, even when a face orientation is thesame as the orientation in a face photograph, when conditions other thana face orientation (for example, a light and shade distribution in animage, the area of a recognizable part, or the like) are significantlydifferent, normal matching may not be performed. In contrast, in the 3Dmatching mode, it is possible to perform matching by flexibly addressinga change in conditions such as the orientation of a face, a lightingcondition, and the presence or absence of an attachment (a worn item).On the contrary, the 3D matching mode has more variations of the faceorientation or the like than a two-dimensional image, and there is aconcern of a reduction in the processing speed and the processingcapacity.

Accordingly, the matching mode select unit 14 determines an imagesuitable for matching out of a two-dimensional and a three-dimensionalregistered image in accordance with the degree to which a face can berecognized in the two-dimensional image and selects a matching mode.Note that the degree to which a face can be recognized may be determinedby the face orientation of a person, the light and shade distribution inan image, the type of a recognizable part, the recognizable area, thepresence or absence of an attachment (an accessory), or the like. In thepresent example embodiment, the degree is determined by the faceorientation of a person.

The matching mode select unit 14 in the present example embodimentselects a type of a registered image based on the face orientation of amatching target calculated in the image analysis unit 13. For example,the matching mode select unit 14 selects a two-dimensional registeredimage when the angle of the face orientation of a matching target in atwo-dimensional image relative to a reference orientation is within apredetermined threshold and selects a three-dimensional registered imagewhen the angle exceeds the threshold. The threshold of the angle in thepresent example embodiment is 30 degrees in the vertical direction andthe horizontal direction (15 degrees each in the upward direction, thedownward direction, the left direction, and the right direction), andthe reference orientation corresponds to the front direction.

FIG. 4A and FIG. 4B are diagrams illustrating a positional relationshipbetween a face orientation of a matching target and the image capturingapparatus 20 in the present example embodiment. In this example, thehorizontal direction is represented by an X-axis and a Y-axis orthogonalto each other, and the perpendicular direction is represented by aZ-axis orthogonal to both the X-axis and the Y-axis. Further, thecapturing direction of the image capturing apparatus 20 matches theaxial direction of the X-axis.

FIG. 4A illustrates a state of a head part of a matching target whenviewed from the side. Here, with reference to a state where the matchingtarget looks at the image capturing apparatus 20 from the front, theface orientation of the matching target is inclined by an angle θ in theperpendicularly upward direction (the Z-axis positive direction in FIG.4A). In contrast, FIG. 4B illustrates a state of the head part of thematching target when viewed from the overhead side. Here, with referenceto a state where the matching target looks at the image capturingapparatus 20 from the front, the face orientation of the matching targetis inclined by an angle θ in the left direction (the Y-axis negativedirection in FIG. 4B).

A matching mode select unit 14 of the present example embodiment uses atwo-dimensional registered image for matching when the angle θ of theface orientation of the matching target relative to the front directionis within a threshold (15 degrees) in both FIG. 4A and FIG. 4B. Thematching mode select unit 14 then uses a three-dimensional registeredimage for matching when the angle θ exceeds the threshold in either oneof FIG. 4A and FIG. 4B. The same applies to a case where the faceorientation of the matching target is the perpendicularly downwarddirection and a right direction.

The matching unit 15 matches a two-dimensional image of a matchingtarget with a registered image group including three-dimensionalregistered images of a plurality of registrants or a registered imagegroup including two-dimensional registered images of a plurality ofregistrants based on a matching mode selected by the matching modeselect unit 14 and calculates a similarity degree between the matchingtarget and registrants on a registrant basis.

The matching unit 15 matches a two-dimensional image with atwo-dimensional registered image when the matching mode is a 2D mode.Further, the matching unit 15 matches a two-dimensional image with athree-dimensional registered image when the matching mode is a 3D mode.In such a case, the matching unit 15 first adjusts the face orientationof the registrant in the three-dimensional registered image to beconsistent with the face orientation of a person in the two-dimensionalimage. Next, the matching unit 15 creates a two-dimensional projectionimage of the front from the three-dimensional registered image in whichthe face orientation has been adjusted. The matching unit 15 thenmatches the projection image with the two-dimensional image.

The display information generation unit 16 extracts a plurality ofcandidates from a plurality of registrants based on the similaritydegree obtained by matching of a two-dimensional image of a matchingtarget with a registered image group including three-dimensionalregistered images of a plurality of registrants and generates displayinformation used for displaying the extracted candidates in order inaccordance with the similarity degree. The display informationgeneration unit 16 displays a candidate display window based on thegenerated display information on a display.

The composite image generation unit 17 superimposes, on atwo-dimensional image of a matching target, a three-dimensionalregistered image of a person selected from a plurality of candidates bya user operation on the candidate display window and thereby generates acomposite image. The composite image generation unit 17 displays aperson check window including the generated composite image on thedisplay.

FIG. 5 is a block diagram illustrating a hardware configuration exampleof the image matching apparatus 10 in the present example embodiment.The image matching apparatus 10 has a central processing unit (CPU) 151,a random access memory (RAM) 152, a read only memory (ROM) 153, and ahard disk drive (HDD) 154 as a computer that performs operation,control, and storage. Further, the image matching apparatus 10 has acommunication interface (I/F) 155, a display device 156, and an inputdevice 157. The CPU 151, the RAM 152, the ROM 153, the HDD 154, thecommunication I/F 155, the display device 156, and the input device 157are connected to each other via a bus 158. Note that the display device156 and the input device 157 may be connected to the bus 158 via a drivedevice (not illustrated) used for driving these devices.

The CPU 151 is a processor having a function of performing apredetermined operation in accordance with a program stored in the ROM153, the HDD 154, or the like and controlling each component of theimage matching apparatus 10. The RAM 152 is formed of a volatile storagemedium and provides a temporal memory area necessary for the operationof the CPU 151. The ROM 153 is formed of nonvolatile storage medium andstores necessary information such as a program used for the operation ofthe image matching apparatus 10. The HDD 154 is a storage device that isformed of a nonvolatile storage medium and stores data necessary forprocessing, an operation program of the image matching apparatus 10, orthe like.

The communication I/F 155 is a communication interface based on thespecification such as Ethernet (registered trademark), Wi-Fi (registeredtrademark), 4G, or the like, which is a module used for communicatingwith other apparatuses. The display device 156 is a liquid crystaldisplay, an OLED display, or the like and is used for displaying animage, a text, an interface, or the like. The input device 157 is akeyboard, a pointing device, or the like and is used by the user foroperating the image matching apparatus 10. An example of the pointingdevice may be a mouse, a trackball, a touchscreen, a pen tablet, or thelike. The display device 156 and the input device 157 may be integrallyformed as a touchscreen.

The CPU 151 loads a program stored in the ROM 153, the HDD 154, or thelike to the RAM 152 and executes the program. Thereby, the CPU 151implements the functions of the image acquisition unit 12, the imageanalysis unit 13, the matching mode select unit 14, the matching unit15, the display information generation unit 16, the composite imagegeneration unit 17, or the like described above. Furthermore, the CPU151 implements the function of the storage unit 11 by controlling theHDD 154.

Note that the hardware configuration illustrated in FIG. 5 is anexample, and a device other than the above may be added, or some of thedevices may not be provided. Further, some of the devices may bereplaced with another device having the same function. Furthermore, apart of the function of the present example embodiment may be providedby another device via the network 40, the function of the presentexample embodiment may be implemented by being distributed in aplurality of devices. For example, the HDD 154 may be replaced with asolid state drive (SSD) with a semiconductor memory or may be replacedwith a cloud storage.

Next, the operation of the image matching apparatus 10 will be describedwith reference to FIG. 6 to FIG. 9 . FIG. 6 is a flowchart illustratingan example of a matching process in the present example embodiment. Forexample, this process is repeatedly performed from an instruction tostart performing a matching process by a user operation to aninstruction to stop performing the matching process.

First, the CPU 151 (the image acquisition unit 12) of the image matchingapparatus 10 acquires a two-dimensional image of a matching target fromthe image capturing apparatus 20 or the reading apparatus 30 (stepS101). Next, the CPU 151 (the image analysis unit 13) analyzes thetwo-dimensional image to detect the face orientation of the matchingtarget (step S102).

Next, the CPU 151 (the matching mode select unit 14) determines whetheror not the detected face orientation is within 30 degrees (threshold)with respect to the front (step S103). In this step, if the CPU 151 (thematching mode select unit 14) determines that the face orientation iswithin 30 degrees with respect to the front (step S103: YES), the CPU151 (the matching mode select unit 14) selects the 2D matching mode as amatching mode (step S104), and the process proceeds to step S106.

In contrast, if the CPU 151 (the matching mode select unit 14)determines that the face orientation is not within 30 degrees withrespect to the front (step S103: NO), the CPU 151 (the matching modeselect unit 14) selects the 3D matching mode as a matching mode (stepS105), and the process proceeds to step S106.

In step S106, the CPU 151 (the matching unit 15) matches thetwo-dimensional image of the matching target with a registered imagegroup of a plurality of registrants based on the matching mode selectedby the matching mode select unit 14. Thereby, the CPU 151 (the matchingunit 15) calculates a similarity degree between the matching target andregistrants on a registrant basis.

The CPU 151 (the display information generation unit 16) then extracts aplurality of candidates from the plurality of registrants and generatesdisplay information used for displaying the extracted candidates inorder in accordance with the similarity degree based on the similaritydegree obtained by matching the two-dimensional image of the matchingtarget with a registered image group and, in response, displays thedisplay information as a candidate display window on the display device156 (step S107).

FIG. 7 is a diagram illustrating an example of the candidate displaywindow in the present example embodiment. The candidate display windowin FIG. 7 displays a face image of a matching target, capturing time, acapturing place, an extraction condition, and a sort condition in theupper field. While the similarity degree is set as an initial setting,for example, the sort condition may be designated by a user operationfrom data items such as a gender or an address of a candidate.

Further, in a lower field, candidates extracted from a plurality ofregistrants by matching with the face image of the matching target aredisplayed in a form of a list in order of similarity degree. The displayinformation on the list includes the rank of the similarity degree, thesimilarity degree, the age and the address of the candidate in additionto the face image (projection image) obtained by projection ofthree-dimensional registered image of the candidate. Further, the faceorientation of a candidate in a face image has been corrected to beconsistent with the face orientation of the matching target, whichfacilitates comparison of images of the matching target with acandidate.

FIG. 8 is a flowchart illustrating an example of a generation process ofa composite image in the present example embodiment. This process isperformed in association with the display process of the candidatedisplay window in the display device 156.

First, the CPU 151 (the composite image generation unit 17) of the imagematching apparatus 10 determines whether or not there is selection of acandidate by a user operation on the candidate display window (stepS201). In this step, if the CPU 151 (the composite image generation unit17) determines that there is selection of a candidate (step S201: YES),the process proceeds to step S202. In contrast, if the CPU 151 (thecomposite image generation unit 17) determines that there is noselection of a candidate (step S201: NO), the process proceeds to stepS206.

In step S202, the CPU 151 (the composite image generation unit 17)acquires a three-dimensional registered image of the candidate from theHDD 154 (the storage unit 11) based on a registrant ID related to thecandidate (step S202).

Next, the CPU 151 (the composite image generation unit 17) adjusts theface orientation and the size of the image of the candidate in thethree-dimensional registered image to be consistent with the faceorientation and the size of an image of a matching target in atwo-dimensional image (step S203) and then generates a composite imageof the matching target and the candidate (step S204).

Next, the CPU 151 (the composite image generation unit 17) displays, onthe display device 156, a person check window including the compositeimage of the two-dimensional image of the matching target and thethree-dimensional registered image of the candidate selected by a useroperation (step S205).

FIG. 9 is a diagram illustrating an example of a person check window inthe present example embodiment. In this example, in the upper field inthe person check window, a two-dimensional image IMG_01 of a matchingtarget and a three-dimensional registered image IMG_02 of a candidateselected from a list are juxtaposed and displayed. Further, in the lowerfield in the window, four types of composite images IMG_03 to IMG_06generated from the two-dimensional image IMG_01 and thethree-dimensional registered image IMG_02 of the upper field arearranged and displayed.

The composite images IMG_03 and IMG_04 are horizontal wipe images inwhich the two-dimensional image IMG_01 and the three-dimensionalregistered image IMG_02 are divided horizontally into two and combined,respectively. Similarly, the composite images IMG_05 and IMG_06 arevertical wipe images in which the two-dimensional image IMG_01 and thethree-dimensional registered image IMG_02 are divided vertically intotwo and combined, respectively. When each composite image is generated,the three-dimensional registered image IMG_02 of the candidate has beenadjusted to be consistent with the face orientation and the positioncoordinates of a face feature part (an eye, a nose, a mouth, or thelike) within the image in the face image IMG_01 of the matching target.

Note that the type of composite images is not limited to only thevertical and horizontal wipe images. For example, the face image of anyone of a matching target and a candidate may be converted into asemi-transparent image, and a composite image in which thesemi-transparent image is superimposed on the other face image may begenerated.

In step S206, the CPU 151 (the composite image generation unit 17)determines whether or not the end button of a check operation is pressedby a user operation on a check window (step S206). In this step, if theCPU 151 (the composite image generation unit 17) determines that the endbutton is pressed (step S206: YES), the process ends. In contrast, ifthe CPU 151 (the composite image generation unit 17) determines that theend button is not pressed (step S206: NO), the process returns to stepS201, and the process of steps S201 to S206 is repeated until the endbutton is pressed.

As described above, the image matching apparatus 10 in the presentexample embodiment selects, from one of a two-dimensional registeredimage and a three-dimensional registered image, a type of a registeredimage used for matching based on an analysis result of a two-dimensionalimage prior to matching of the two-dimensional image (captured image)acquired from the image capturing apparatus 20 or the reading apparatus30 with a registered image group. Thereby, the image matching apparatus10 can perform a matching process in a matching mode suitable tomatching of the two-dimensional image of the matching target. When the2D matching mode is selected, a fast and accurate matching process isenabled.

On the other hand, when the 3D matching mode is selected because theface orientation of a matching target in the two-dimensional imageexceeds a threshold, the face orientation of a candidate in thethree-dimensional registered image is adjusted to be consistent with theface orientation of the matching target in the two-dimensional image.Thus, the image matching apparatus 10 can flexibly addresstwo-dimensional images captured in various capturing angles to perform amatching process.

Further, the image matching apparatus 10 in the present exampleembodiment lists a plurality of registrants matched with a matchingtarget in order of the similarity degree. Thereby, the user can proceedwith a check operation while sequentially selecting candidates having ahigh similarity degree and thus can efficiently perform a visual checkoperation.

Further, the image matching apparatus 10 in the present exampleembodiment displays a check window including a composite image in whicha face image of a matching target and a face image of a candidateselected from a list are composed. In the composite image, the faceimage of a candidate is corrected to cause the face orientation and thesize thereof to be consistent with the face image of the matchingtarget. This enables the user to easily check whether or not thematching target and the candidate are the same person by referencing thecomposite image.

Note that, while the image matching apparatus 10 in the present exampleembodiment is supposed to pre-store both the two-dimensional registeredimage and the three-dimensional registered image in a database (thestorage unit 11) as registrant information, a configuration that storesonly the three-dimensional registered image may be possible. In such acase, when the 2D matching mode is selected, two-dimensional matchingimage may be generated from the three-dimensional registered image andused in a matching process.

Further, while a case where the age and the address that are attributeinformation of a candidate are displayed in addition to a similaritydegree has been described in the candidate display window illustrated inFIG. 7 , displayed attribute information is not limited thereto. Forexample, attribute information such as the gender or the birthplace of acandidate may be displayed. Furthermore, the display informationgeneration unit 16 may change the order of candidates in displayinformation based on a data item selected by a user operation out of aplurality of data items included in attribute information.

Second Example Embodiment

The image matching system 1 in a second example embodiment will bedescribed below. Note that a reference common to the reference providedin the drawings of the first example embodiment indicates the samecomponent. The description of features common to the first exampleembodiment will be omitted, and different features will be described indetail.

The image matching system 1 in the present example embodiment isdifferent from the first example embodiment in that the image matchingapparatus 10 (the composite image generation unit 17) further has afunction of performing an editing process for changing the appearance ofa registrant on a three-dimensional registered image when superimposinga two-dimensional image of a certain person (a matching target) on athree-dimensional registered image of the registrant to generate acomposite image.

Here, specific examples of an editing process for the appearance may be(A) to (F) or the like below.

(A) A process for adjusting a hair growth part, an amount, a shape(hairstyle or the like), a color of body hair (hair of head, beard,eyebrow, or the like).

(B) A process for adjusting the presence or absence of an attachment(glasses, a cap, a mask, a piercing jewelry, a tattoo, or the like).

(C) A process for simulating a change of the type or the extent offacial expression.

(D) A process for simulating a change of a body shape or the presence orabsence of a scar.

(E) A process for simulating a change over the years of an aging degreeof a face.

(F) A process for adjusting an influence degree of environmental light.

For (A) to (F) described above, in response to detecting a difference inthe appearance between a two-dimensional image and a three-dimensionalregistered image, the composite image generation unit 17 of the presentexample embodiment performs an editing process for causing theappearance of one of the images to be closer to the appearance of theother image.

FIG. 10 is a flowchart illustrating an example of a generation processof a composite image in the present example embodiment. For example,this process is performed when a candidate is selected based on a useroperation on the candidate display window displayed when a matchingprocess is performed as with the first example embodiment.

First, the CPU 151 (the composite image generation unit 17) acquires atwo-dimensional image of a matching target (step S301). Next, the CPU151 (the composite image generation unit 17) acquires athree-dimensional registered image of a candidate from the HDD 154 (thestorage unit 11) based on a registrant ID related to the candidate (stepS302).

Next, the CPU 151 (the composite image generation unit 17) adjusts thepositional relationship of the face orientation and the feature portionof the candidate in the three-dimensional registered image to beconsistent with the positional relationship between the face orientationand the feature portion of the matching target in the two-dimensionalimage (step S303).

Next, the CPU 151 (the composite image generation unit 17) compares thetwo-dimensional image with the three-dimensional registered image anddetermines whether or not a hair growth part (the head or the chin), ahair growth amount and shape, or the like is different between the twoimages beyond a predetermined tolerance range (step S304).

Here, if the CPU 151 (the composite image generation unit 17) determinesthat a hair growth part, a hair growth amount and shape, or the like isdifferent beyond the predetermined tolerance range (step S304: YES), theCPU 151 (the composite image generation unit 17) then adjusts the hairgrowth part or the like of the candidate in the three-dimensionalregistered image to be consistent with the hair growth part or the likeof the matching target in the two-dimensional image (step S305), and theprocess proceeds to step S306. The hair growth part or the hair growthamount and shape of the candidate in the three-dimensional registeredimage may be automatically selected from predetermined templatesregarding hair growth parts.

In contrast, if the CPU 151 (the composite image generation unit 17)determines that a hair growth part or the like is not different beyondthe predetermined tolerance range between the two-dimensional image andthe three-dimensional registered image (step S304: NO), the processproceeds to step S306.

In step S306, the CPU 151 (the composite image generation unit 17)determines whether or not there is a difference in an attachment such asglasses or a mask between the two-dimensional image and thethree-dimensional registered image. Here, if the CPU 151 (the compositeimage generation unit 17) determines that there is a difference in anattachment (step S306: YES), the CPU 151 (the composite image generationunit 17) causes the candidate in the three-dimensional registered imageto wear an attachment in accordance with the attachment of the matchingtarget in the two-dimensional image (step S307), and the processproceeds to step S308.

For example, when only the matching target in the two-dimensional imagewears glasses, an editing process is performed to cause the candidate inthe three-dimensional registered image to wear glasses similar to theglasses of the matching target. The similar glasses may be automaticallyselected from predetermined templates regarding attachments. On thecontrary, when only the candidate in the three-dimensional image wearsglasses, an editing process may be performed to cause the candidate inthe three-dimensional registered image to put off the glasses.

In contrast, if the CPU 151 (the composite image generation unit 17)determines that there is no difference in an attachment between thetwo-dimensional image and the three-dimensional registered image (stepS306: NO), the process proceeds to step S308. Note that, while it issufficient to determine a difference in the attachment in accordancewith classification of articles, attachments having differentappearances, such as typical glasses and sunglasses, may be determinedas different articles.

In step S308, the CPU 151 (the composite image generation unit 17)determines whether or not, at the current date and time, a predeterminedperiod has elapsed from an image capture date (registration date) of thethree-dimensional registered image. Here, if the CPU 151 (the compositeimage generation unit 17) determines that the predetermined period haselapsed from the image capture date (step S308: YES), the CPU 151 (thecomposite image generation unit 17) performs an editing process forsimulating a change over the years of the candidate (step S309), and theprocess proceeds to step S310.

For example, if a period of 10 years has elapsed from an image capturedate of a three-dimensional image to the current date and time, anediting process for simulating a 10-year aged state of the candidate isperformed. On the contrary, if a capture date and time of a capturedimage read by the reading apparatus 30 is older than an imageregistration date of a three-dimensional registered image, an editingprocess for simulating a state where the candidate of thethree-dimensional registered image is rejuvenated may be performed. Thatis, the composite image generation unit 17 performs an editing processfor simulating a change over the years of the aging degree on one of atwo-dimensional image and a three-dimensional registered image based onattribute information of the two-dimensional image and thethree-dimensional registered image.

In contrast, if the CPU 151 (the composite image generation unit 17)determines that the predetermined period has not yet elapsed from theimage capture date (step S308: NO), the process proceeds to step S310.The predetermined period may be set to any period and may be set to alength with which a significant change in the appearance may occur dueto elapsed time.

In step S310, if the CPU 151 (the composite image generation unit 17)determines whether or not the type of facial expression of a person isdifferent between the two-dimensional image and the three-dimensionalregistered image. Here, if the CPU 151 (the composite image generationunit 17) determines that the type of facial expression is different(step S310: YES), the CPU 151 (the composite image generation unit 17)adjusts the type of facial expression of the candidate in thethree-dimensional registered image to be consistent with the type of thefacial expression of the matching target in the two-dimensional image(step S311), and the process proceeds to step S312.

For example, when the type of facial expression of a matching target ina two-dimensional image is “anger” and the type of facial expression ofa candidate in a three-dimensional registered image is “expressionless”,an editing process for simulating a state where the type of the facialexpression of the candidate is changed to “anger” as with the matchingtarget is performed. Note that an editing process may be performed so asto determine not only the type of facial expression but also the extentof facial expression to have the same extent thereof.

Note that, to determine the type of facial expression, a step ofestimating facial expression from a separate image may be provided.Facial expression may be determined by using an external device, and aresult thereof may be used. A scheme for determining facial expressionis not limited.

In contrast, if the CPU 151 (the composite image generation unit 17)determines that the type of facial expression is not different betweenthe two images (step S310: NO), the process proceeds to step S312.

Then, the CPU 151 (the composite image generation unit 17) generatescomposite images of the matching target and the candidate (step S312)and, in response, displays the person check window including thecomposite images on the display device 156 (step S313).

Note that the order of four determination processes in FIG. 10 (stepsS304 to S305/steps S306 to S307/steps S308 to S309/steps S310 to S311)is not limited thereto and may be replaced with any order. Further, thesame effect and advantage may be obtained by performing these processesin parallel.

Next, specific examples of the editing process in the present exampleembodiment will be described based on FIG. 11A to FIG. 18D.

(1) Editing Process with Respect to Hair Growth Part

FIG. 11A illustrates a two-dimensional image IMG_11 of a matchingtarget, and FIG. 11B illustrates a three-dimensional registered imageIMG_12 of a candidate. As illustrated in the two-dimensional imageIMG_11, the hairstyle of the candidate is “longhair”. As illustrated inthe three-dimensional registered image IMG_12, however, the hairstyle ofthe matching target is “shorthair” and is different from that of thematching target.

Further, FIG. 11C illustrates a horizontally composed image IMG_13 ofthe two-dimensional image IMG_11 of the matching target and thethree-dimensional registered image IMG_12 of the candidate, and FIG. 11Dillustrates a vertically composed image IMG_14 thereof.

With reference to the composite image IMG_13 and the composite imageIMG_14, it can be seen that positions of face feature portions (eyes, anose, a mouth, and the like) and the contour of the whole face are thesame between the images. However, the hairstyle is significantlydifferent between the two images. Thus, comparison of the compositeimage IMG_13 and the composite image IMG_14 with the two-dimensionalimage IMG_11 and the three-dimensional registered image IMG_12 may notbe easy.

In contrast, FIG. 12A illustrates a two-dimensional image IMG_21 of thematching target, and FIG. 12B illustrates a three-dimensional editedimage IMG_22 obtained by performing an editing process on thethree-dimensional registered image IMG_12 (see FIG. 11B) of thecandidate. As illustrated in IMG_21, the hairstyle of the matchingtarget is “longhair”. On the other hand, as illustrated in thethree-dimensional edited image IMG_22, the hairstyle of the candidate ischanged to “longhair” as with the matching target.

Further, FIG. 12C illustrates a horizontally composed image IMG_23 ofthe two-dimensional image IMG_21 of the matching target and thethree-dimensional edited image IMG_22 of the candidate, and FIG. 12Dillustrates a vertically composed image IMG_24 thereof.

With reference to the composite image IMG_23 and the composite imageIMG_24, it can be seen that positions of face feature portions (eyes, anose, a mouth, and the like) and the contour of the whole face are thesame between the images. Furthermore, since the hairstyle is unified insubstantially the same manner between the two images to be composed, thecomposite image IMG_23 and the composite image IMG_24 are images havinga unified look in the horizontal direction and the vertical direction.Thus, comparison of the composite image IMG_23 and the composite imageIMG_24 with the two-dimensional image IMG_21 and the three-dimensionaledited image IMG_22 is easier than in the case of FIG. 11A to FIG. 11D.

(2) Editing Process with Respect to Attachment

FIG. 13A illustrates a two-dimensional image IMG_31 of a matchingtarget, and FIG. 13B illustrates a three-dimensional registered imageIMG_32 of a candidate. As illustrated in the two-dimensional imageIMG_31, the matching target wears the attachment “glasses”. Asillustrated in the three-dimensional registered image IMG_32, however,the candidate does not wear “glasses” and has a different hairstyle.

Further, FIG. 13C illustrates a horizontally composed image IMG_33 ofthe two-dimensional image IMG_31 of the matching target and thethree-dimensional registered image IMG_32 of the candidate, and FIG. 13Dillustrates a vertically composed image IMG_34 thereof.

With reference to the composite image IMG_33 and the composite imageIMG_34, one of the two images to be composed has the attachment and hairbut the other does not. Thus, comparison of the composite image IMG_33and the composite image IMG_34 with the two-dimensional image IMG_31 andthe three-dimensional registered image IMG_32 may not be easy.

In contrast, FIG. 14A illustrates a two-dimensional image IMG_41 of thematching target, and FIG. 14B illustrates a three-dimensional editedimage IMG_42 obtained by performing an editing process on thethree-dimensional registered image IMG_32 (see FIG. 13B) of thecandidate. As illustrated in IMG_41, the matching target wears theattachment “glasses”. On the other hand, as illustrated in thethree-dimensional edited image IMG_42, the candidate wears theattachment “glasses” as with the matching target and has a changedhairstyle.

Further, FIG. 14C illustrates a horizontally composed image IMG_43 ofthe two-dimensional image IMG_41 of the matching target and thethree-dimensional edited image IMG_42 of the candidate, and FIG. 14Dillustrates a vertically composed image IMG_44 thereof.

With reference to the composite image IMG_43 and the composite imageIMG_44, since the presence or absence of an attachment and the hairstyleare unified between the two images to be composed, the composite imageIMG_43 and the composite image IMG_44 are images having a unified lookin the horizontal direction and the vertical direction. Thus, comparisonof the composite image IMG_43 and the composite image IMG_44 with thetwo-dimensional image IMG_41 and the three-dimensional edited imageIMG_42 is easier than in the case of FIG. 13A to FIG. 13D.

(3) Editing Process with Respect to Change in Facial Expression

FIG. 15A illustrates a two-dimensional image IMG_51 of a matchingtarget, and FIG. 15B illustrates a three-dimensional registered imageIMG_52 of a candidate. As illustrated in the two-dimensional imageIMG_51, the facial expression of the matching target is “anger” offrowning. As illustrated in the three-dimensional registered imageIMG_52, however, the facial expression of the candidate has usual facialexpression (“expressionless”) and has a different hairstyle.

Further, FIG. 15C illustrates a horizontally composed image IMG_53 ofthe two-dimensional image IMG_51 of the matching target and thethree-dimensional registered image IMG_52 of the candidate, and FIG. 15Dillustrates a vertically composed image IMG_54 thereof.

With reference to the composite image IMG_53 and the composite imageIMG_54, the facial expression and the hairstyle of persons aresignificantly different between the two images to be composed. Thus,comparison of the composite image IMG_53 and the composite image IMG_54with the two-dimensional image IMG_51 and the three-dimensionalregistered image IMG_52 may not be easy.

In contrast, FIG. 16A illustrates a two-dimensional image IMG_61 of thematching target, and FIG. 16B illustrates a three-dimensional editedimage IMG_62 obtained by performing an editing process on thethree-dimensional registered image IMG_52 (see FIG. 15B) of thecandidate. As illustrated in the two-dimensional image IMG_61, thefacial expression of the matching target is a facial expression of“anger”. On the other hand, as illustrated in the three-dimensionaledited image IMG_62, the hairstyle has been changed, and in addition,the facial expression of the candidate has been changed to the facialexpression of “anger” as with the matching target. That is, an editingprocess for simulating a facial expression change is performed.

Further, FIG. 16C illustrates a horizontally composed image IMG_63 ofthe two-dimensional image IMG_61 of the matching target and thethree-dimensional edited image IMG_62 of the candidate, and FIG. 16Dillustrates a vertically composed image IMG_64 thereof.

With reference to the composite image IMG_63 and the composite imageIMG_64, since the facial expression and the hairstyle of persons areunified between the two images to be composed, the composite imageIMG_63 and the composite image IMG_64 are images having a unified lookin the horizontal direction and the vertical direction. Thus, comparisonof the composite image IMG_63 and the composite image IMG_64 with thetwo-dimensional image IMG_61 and the three-dimensional edited imageIMG_62 is easier than in the case of FIG. 15A to FIG. 15D.

(4) Editing Process with Respect to Change Over the Years

FIG. 17A illustrates a two-dimensional image IMG_71 of a matchingtarget, and FIG. 17B illustrates a three-dimensional registered imageIMG_72 of a candidate. As illustrated in the two-dimensional imageIMG_71, the matching target is an elderly man. On the other hand, thecandidate illustrated in the three-dimensional registered image IMG_72is a man in his thirties (see FIG. 3 ) and has a different hairstyle.

FIG. 17C illustrates a horizontally composed image IMG_73 of thetwo-dimensional image IMG_71 of the matching target and thethree-dimensional registered image IMG_72 of the candidate, and FIG. 17Dillustrates a vertically composed image IMG_74 thereof.

With reference to the composite image IMG_73 and the composite imageIMG_74, the aging degree and the hairstyle of persons are significantlydifferent between the two images to be composed. Thus, comparison of thecomposite image IMG_73 and the composite image IMG_74 with thetwo-dimensional image IMG_71 and the three-dimensional registered imageIMG_72 may not be easy.

In contrast, FIG. 18A illustrates a two-dimensional image IMG_81 of thematching target, and FIG. 18B illustrates a three-dimensional editedimage IMG_82 obtained by performing an editing process on thethree-dimensional registered image IMG_72 (see FIG. 17B) of thecandidate. As illustrated in the two-dimensional image IMG_81, thematching target is an elderly man in his sixties or older. On the otherhand, as illustrated in the three-dimensional edited image IMG_82, thecandidate not only has a changed hairstyle but also looks aged foraround 30 years as with the matching target. That is, an editing processfor simulating a change over the years is performed.

FIG. 18C illustrates a horizontally composed image IMG_83 of thetwo-dimensional image IMG_81 of the matching target and thethree-dimensional edited image IMG_82 of the candidate, and FIG. 18Dillustrates a vertically composed image IMG_84 thereof.

With reference to the composite image IMG_83 and the composite imageIMG_84, since the aging degree and the hairstyle of persons are unifiedbetween the two images to be composed, the composite image IMG_83 andthe composite image IMG_84 are images having a unified look in thehorizontal direction and the vertical direction. Thus, comparison of thecomposite image IMG_83 and the composite image IMG_84 with thetwo-dimensional image IMG_81 and the three-dimensional edited imageIMG_82 is easier than in the case of FIG. 17A to FIG. 17D.

Note that, while four types of editing processes have been described,the type of editing processes is not limited thereto. For example, anediting process for causing the influence degree of environmental light,wearing makeup or not, or the like to be the same between a matchingtarget and a candidate may be performed.

As described above, the image matching apparatus 10 in the presentexample embodiment performs an editing process for changing theappearance of a registrant on a three-dimensional registered image whensuperimposing a two-dimensional image of a matching target on thethree-dimensional registered image of the registrant (the candidate) togenerate a composite image. For example, even when the similarity degree(the matching score) calculated in an image matching process is high, aperson having a different visual impression may be extracted as acandidate due to various factors such as the presence or absence of anattachment, facial expression of a person, aging, or the like. Even insuch a case, by changing the appearance of a registrant to be consistentwith the matching target, it is possible to generate a composite imagehaving a unified look as a whole. As a result, the user can effectivelyperform a visual check operation based on the composite image.

Further, in the present example embodiment, an editing process isautomatically performed based on a comparison result between atwo-dimensional image and a three-dimensional image. This enables theuser to obtain a composite image without performing a designationoperation.

Note that, while the case where an editing process is performed on athree-dimensional registered image of a registrant to be consistent witha two-dimensional image of a matching target has been described in thepresent example embodiment, an editing process (for example, edition toadd “glasses” or “beard” or the like) may be performed on atwo-dimensional image of a matching target to be consistent with athree-dimensional registered image. Further, an editing process may beperformed on a predetermined image region at time after a compositeimage is generated without being limited to at time before thegeneration of a composite image. That is, the image matching apparatus10 may perform an editing process for changing the appearance of aperson or a registrant on at least one of a two-dimensional image, athree-dimensional registered image, and a composite image.

Further, while a composite image is generated between a person selectedfrom a plurality of registrants listed by performing a matching processand a matching target as with the first example embodiment in thepresent example embodiment, such listing may not be performed. Forexample, a registrant to be matched with a matching target may bedesignated by a user operation, and a composite image may be generatedfrom a two-dimensional image and a three-dimensional edited image.Further, the composite image generation unit 17 may be configured toautomatically perform generation and editing processes of a compositeimage in descending order of similarity degree without requiring aselection operation performed by a user.

Furthermore, the composite image generation unit 17 may perform anediting process for an item selected from a plurality of target itemsregarding the appearance based on a user operation. For example, whenonly “presence or absence of attachment” is designated by a useroperation, it is possible to perform an editing process on only theattachment without taking into consideration of a change in facialexpression a person, a change in environmental light, or the like. Thisenables the user to avoid performing an unnecessary editing process.Further, the composite image generation unit 17 may be configured toperform an editing process again when an edition menu is again selectedby a user operation.

Third Example Embodiment

The image matching system 1 in a third example embodiment will bedescribed below. Note that a reference common to the reference providedin the drawings of the first example embodiment indicates the samecomponent. The description of features common to the first exampleembodiment will be omitted, and different features will be described indetail.

The matching mode select unit 14 of the present example embodiment isdifferent from that of the first example embodiment in a determinationcondition used for selecting the type of a registered image to be usedfor matching from one of a two-dimensional registered image and athree-dimensional registered image. Specific examples of determinationconditions may be (A) to (F) or the like below.

(A) Face Orientation

The matching mode select unit 14 selects the 2D matching mode when theangle of the face orientation of a matching target relative to the frontdirection is within a predetermined threshold (for example, 30 degrees)as with the first example embodiment. The matching mode select unit 14selects the 3D matching mode when the threshold is exceeded.

(B) Facial Expression Change

The matching mode select unit 14 analyzes a two-dimensional image andselects the 2D matching mode when the change degree of facial expressionof a matching target relative to a usual state (for example, at the timeof expressionlessness) is within a range suitable to the 2D matchingmode (hereafter, referred to as “tolerance range”). The matching modeselect unit 14 selects the 3D matching mode when the change degreeexceeds the tolerance range.

For example, when the facial expression of a matching target is “heartylaugh”, “furious anger”, or the like, it is expected that such facialexpression is significantly different from the facial expression in thetwo-dimensional registered image. Thus, the matching mode select unit 14selects the 3D matching mode. On the contrary, when the facialexpression of a matching target is “expressionless”, “smile”, or thelike, it is expected that such facial expression is close to the facialexpression in the two-dimensional registered image. Thus, the matchingmode select unit 14 selects the 2D matching mode.

Note that, to determine the type of facial expression, a step ofestimating facial expression from a separate image may be provided.Facial expression may be determined by using an external device, and aresult thereof may be used. A scheme for determining facial expressionis not limited.

(C) Influence Degree of Illumination Light (Environmental Light)

The matching mode select unit 14 analyzes a two-dimensional image andselects the 2D matching mode when the influence degree of illuminationlight on a matching target is within a tolerance range of the 2Dmatching mode. The matching mode select unit 14 selects the 3D matchingmode when the influence degree of illumination light exceeds thetolerance range.

For example, when the influence degree of illumination light on amatching target is large and a dark shade appears on a face image, it isexpected that the influence degree of illumination light, that is, acapturing condition is significantly different from that for atwo-dimensional registered image captured under a dimmed environment andthe tolerance range is exceeded. On the other hand, in the 3D matchingmode, since it is possible to cause a condition of a 3D model(three-dimensional registered image) to be consistent with a capturingcondition of a security camera, matching accuracy increases. Thus, thematching mode select unit 14 selects the 3D matching mode. It is alsopreferable to supplement the position of a light source that irradiatesa face, the presence or absence of glasses, or the like by using the 3Dmodel (three-dimensional registered image). On the contrary, when theinfluence degree of illumination light on a matching target is withinthe tolerance range of the 2D matching mode, such an influence degree isclose to the influence degree of lighting in the two-dimensionalregistered image. Thus, the matching mode select unit 14 selects the 2Dmatching mode. Note that there may be a change in light and shade notcaused by environmental light. Specifically, a change in skin color dueto a suntan or the like is assumed. The matching mode select unit 14selects the 3D matching mode when the influence degree including achange in light and shade not caused by environmental light exceeds thetolerance range of the 2D matching mode.

(D) Change Over the Years

The matching mode select unit 14 selects the 2D matching mode when acapturing date recorded as attribute information on a two-dimensionalimage is within a predetermined period from the current date. Thematching mode select unit 14 selects the 3D matching mode when thecapturing date is out of the predetermined period.

For example, when the current date is “Dec. 1, 2018” and a capturingdate of an acquired two-dimensional image is “Oct. 1, 1997”, the elapsedperiod from the capturing date is long, and a significant change in theappearance is expected. Thus, the matching mode select unit 14 selectsthe 3D matching mode with priority. On the contrary, in a case such aswhere a capturing date of a two-dimensional image is the same as orwithin one month from the current date, it is expected that there is nosignificant change in the appearance. Thus, the matching mode selectunit 14 prioritizes and selects the 2D matching mode in which thematching speed is high.

(E) Area and Type of Recognizable Part

The matching mode select unit 14 analyses a two-dimensional image andselects the 2D matching mode when the area and the type of therecognizable part in a face satisfy a matching condition in the 2Dmatching mode. The matching mode select unit 14 selects the 3D matchingmode when the matching condition in the 2D matching mode is notsatisfied. For example, it is preferable to select the 3D matching modewhen there is only an area where a face region of a matching target maynot be recognized due to the presence of another person or an object.Similarly, a matching mode may be selected in accordance with which partof an eye, an ear, a nose, or the like the recognizable part within animage is. For example, the 3D matching mode is selected when only one ofthe eyes of a matching target is included in an acquired face image.

(F) Presence or Absence of Attachment

The matching mode select unit 14 analyzes a two-dimensional image andselects the 2D matching mode when an attachment (glasses, a mask, a cap,or the like) is absent in a face portion of a matching target. Thematching mode select unit 14 selects the 3D matching mode when anattachment is present. For example, when a matching target wearssunglasses or a mask, the matching mode select unit 14 selects the 3Dmatching mode. Note that some type of attachment may be an article thatdoes not affect a face matching. For example, it may be unnecessary totake an attachment such as a piercing jewelry or an earring intoconsideration in selection of a matching mode.

Further, when the 3D matching mode is selected, it is preferable thatthe matching mode select unit 14 of the present example embodimentinstruct the composite image generation unit 17 to perform, on thethree-dimensional image to be matched, an adjustment process (correctionof the face orientation) or an editing process (adjustment of a facialexpression change/influence of environmental light/a change over theyears/an attachment) associated with a condition which a two-dimensionalimage corresponds to out of (A) to (F) described above.

FIG. 19 is a flowchart illustrating an example of a select process of amatching mode in the present example embodiment. This process is toreplace the process between step S102 and step S106 of the flowchartillustrated in FIG. 6 .

In step S102 illustrated in FIG. 6 , in response to the CPU 151 (theimage analysis unit 13) analyzing a two-dimensional image and detectingthe face orientation of a matching target, the CPU 151 (the matchingmode select unit 14) determines whether or not the detected faceorientation is within 30 degrees (threshold) with respect to the front(step S401). Here, if the CPU 151 (the matching mode select unit 14)determines that the face orientation is within 30 degrees with respectto the front (step S401: YES), the process proceeds to step S402.

In contrast, if the CPU 151 (the matching mode select unit 14)determines that the face orientation is not within 30 degrees withrespect to the front (step S401: NO), the process proceeds to step S406.

In step S402, the CPU 151 (the matching mode select unit 14) determineswhether or not a change in facial expression from a usual state(expressionless) is within a tolerance range in the 2D matching mode.Here, if the CPU 151 (the matching mode select unit 14) determines thata change in facial expression from the usual state (expressionless) iswithin the tolerance range in the 2D matching mode (step S402: YES), theprocess proceeds to step S403.

In contrast, if the CPU 151 (the matching mode select unit 14)determines that a change in facial expression from the usual state(expressionless) exceeds the tolerance range in the 2D matching mode(step S402: NO), the process proceeds to step S406.

In step S403, the CPU 151 (the matching mode select unit 14) determineswhether or not the influence degree of illumination light on thematching target is within a tolerance range in the 2D matching mode.Here, if the CPU 151 (the matching mode select unit 14) determines thatthe influence degree of illumination light is within the tolerance rangein the 2D matching mode (step S403: YES), the process proceeds to stepS404.

In contrast, if the CPU 151 (the matching mode select unit 14)determines that the influence degree of illumination light exceeds thetolerance range in the 2D matching mode (step S403: NO), the processproceeds to step S406.

In step S404, the CPU 151 (the matching mode select unit 14) referencesattribute information on the two-dimensional image and determineswhether or not the capturing date of the two-dimensional image is withina predetermined period. Here, if the CPU 151 (the matching mode selectunit 14) determines that the capturing date of the two-dimensional imageis within the predetermined period (step S404: YES), the processproceeds to step S405.

In contrast, if the CPU 151 (the matching mode select unit 14)determines that the capturing date of the two-dimensional image is notwithin the predetermined period (step S404: NO), the process proceeds tostep S406.

In step S405, if the CPU 151 (the matching mode select unit 14) selectsthe 2D matching mode as a matching mode, the process proceeds to stepS106 of FIG. 6 .

In step S406, if the CPU 151 (the matching mode select unit 14) selectsthe 3D matching mode as a matching mode, the process proceeds to stepS106 of FIG. 6 .

Note that the order of steps S401 to S404 in FIG. 19 is not limited tothe above and is interchangeable. Further, the same effect and advantagemay be obtained by performing these processes in parallel.

As described above, the image matching apparatus 10 in the presentexample embodiment selects the type of a registered image used formatching with a two-dimensional image based on a value obtained by imageanalysis of the two-dimensional image and a plurality of determinationconditions. This enables a matching process by using the optimalmatching mode.

Note that the determination condition for selecting a matching mode isnot limited to the above. For example, determination may be made by acondition such as the presence or absence of an attachment, the presenceor absence of a scar and the size thereof, or the like. Further, the 2Dmatching mode, which is superior in a processing speed, may be selectedwhen the number of persons included in a two-dimensional image is large,and the 3D matching mode, which has flexibility for a capturing angle,may be selected when the number of persons is small.

Fourth Example Embodiment

FIG. 20 is a block diagram illustrating the function of the informationprocessing apparatus 100 in a fourth example embodiment. The informationprocessing apparatus 100 has an acquisition unit 110 and a selectionunit 120. The acquisition unit 110 acquires a two-dimensional image of aperson, and a registered image group including two-dimensionalregistered images and three-dimensional registered images of a pluralityof registrants. The selection unit 120 selects a type of a registeredimage to be used for matching from one of the two-dimensional registeredimages and the three-dimensional registered images based on thetwo-dimensional image, prior to the matching the two-dimensional imagewith the registered image group. According to the present exampleembodiment, the speed of the matching process for person images can beincreased.

Modified Example Embodiments

While this disclosure has been described above with reference to theexample embodiments, this disclosure is not limited to the exampleembodiments described above. Various modifications that may beappreciated by those skilled in the art can be made to the configurationand the details of this disclosure within a scope not departing from thespirit of this disclosure. For example, it is to be appreciated that anexample embodiment in which a part of the configuration of any of theexample embodiments is added to another example embodiment or an exampleembodiment in which a part of the configuration of any of the exampleembodiments is replaced with a part of the configuration of anotherexample embodiment is also one of the example embodiments to which thisdisclosure may be applied.

Further, the scope of each of the example embodiments includes aprocessing method that stores, in a storage medium, a program thatcauses the configuration of each of the example embodiments to operateso as to implement the function of each of the example embodimentsdescribed above, reads the program stored in the storage medium as acode, and executes the program in a computer. That is, the scope of eachof the example embodiments also includes a computer readable storagemedium. Further, each of the example embodiments includes not only thestorage medium in which the program described above is stored but alsothe program itself.

As the storage medium, for example, a floppy (registered trademark)disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, amagnetic tape, a nonvolatile memory card, a ROM, or the like can beused. Further, the scope of each of the example embodiments includes anexample that operates on OS to perform a process in cooperation withanother software or a function of an add-in board without being limitedto an example that performs a process by an individual program stored inthe storage medium.

The whole or part of the example embodiments disclosed above can bedescribed as, but not limited to, the following supplementary notes.

(Supplementary Note 1)

An information processing apparatus comprising:

-   -   an acquisition unit that acquires a two-dimensional image of a        person, and a registered image group including two-dimensional        registered images and three-dimensional registered images of a        plurality of registrants; and    -   a selection unit that selects a type of a registered image to be        used for matching from one of the two-dimensional registered        images and the three-dimensional registered images based on the        two-dimensional image, prior to the matching the two-dimensional        image with the registered image group.        (Supplementary Note 2)

The information processing apparatus according to supplementary note 1,wherein the selection unit selects the type of the registered imagebased on the face orientation of the person in the two-dimensionalimage.

(Supplementary Note 3)

The information processing apparatus according to supplementary note 2,wherein the selection unit selects the two-dimensional registered imagewhen the angle formed by the face orientation of the person and thereference direction in the two-dimensional image is not less than apredetermined threshold value, and selects the three-dimensionalregistered image when the angle exceeds the threshold value.

(Supplementary Note 4)

The information processing apparatus according to supplementary note 2or 3 further comprising a matching unit that matches the two-dimensionalimage with the registered image group,

-   -   wherein the matching unit adjusts the face orientation of the        registrant in the three-dimensional registered image to be        consistent with the face orientation of a person in the        two-dimensional image when the three-dimensional registered        image is used for the matching.        (Supplementary Note 5)

The information processing apparatus according to supplementary note 1,wherein the selection unit selects the type of the registered imagebased on the relationship between the person and the illumination lightin the two-dimensional image.

(Supplementary Note 6)

The information processing apparatus according to supplementary note 1,wherein the selection unit selects the type of the registered imagebased on an expression of the person in the two-dimensional image.

(Supplementary Note 7)

The information processing apparatus according to supplementary note 1,wherein the selection unit selects the type of the registered imagebased on an elapsed period from the capturing date of thetwo-dimensional image.

(Supplementary Note 8)

The information processing apparatus according to any one ofsupplementary notes 1 to 7 further comprising a display informationgeneration unit that extracts a plurality of candidates from theplurality of registrants and generates display information used fordisplaying the extracted candidates in order in accordance with thesimilarity degree based on the similarity degree obtained by matchingthe two-dimensional image of the matching target with a registered imagegroup.

(Supplementary Note 9)

The information processing apparatus according to supplementary note 8further comprising a composite image generation unit that superimposesthe three-dimensional registered image of a person selected from aplurality of candidates by a user operation on the two-dimensional imageand generates a composite image.

(Supplementary Note 10)

The information processing apparatus according to supplementary note 9,wherein the composite image generation unit performs an editing processfor changing the appearance of the person or the registrant on at leastone of the two-dimensional image, the three-dimensional registeredimage, and the composite image when generating the composite image.

(Supplementary Note 11)

An information processing method comprising:

-   -   acquiring a two-dimensional image of a person, and a registered        image group including two-dimensional registered images and        three-dimensional registered images of a plurality of        registrants; and    -   selecting a type of a registered image to be used for matching        from one of the two-dimensional registered images and the        three-dimensional registered images based on the two-dimensional        image, prior to the matching the two-dimensional image with the        registered image group.        (Supplementary Note 12)

A storage medium in which a program is stored, the program that causes acomputer to perform:

-   -   acquiring a two-dimensional image of a person, and a registered        image group including two-dimensional registered images and        three-dimensional registered images of a plurality of        registrants; and    -   selecting a type of a registered image to be used for matching        from one of the two-dimensional registered images and the        three-dimensional registered images based on the two-dimensional        image, prior to the matching the two-dimensional image with the        registered image group.

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2018-244593, filed on Dec. 27, 2018, thedisclosure of which is incorporated herein in its entirety by reference.

REFERENCE SIGNS LIST

-   -   1 image matching system    -   10 image matching apparatus    -   11 storage unit    -   12 image acquisition unit    -   13 image analysis unit    -   14 matching mode select unit    -   15 matching unit    -   16 display information generation unit    -   17 composite image generation unit    -   20 image capturing apparatus    -   30 reading apparatus    -   40 network    -   100 information processing apparatus    -   110 acquisition unit    -   120 selection unit    -   151 CPU    -   152 RAM    -   153 ROM    -   154 HDD    -   155 communication I/F    -   156 display device    -   157 input device    -   158 bus

What is claimed is:
 1. An information processing apparatus comprising:an acquisition unit that acquires a two-dimensional image of a person,and a registered image group including two-dimensional registered imagesand three-dimensional registered images of a plurality of registrants,wherein the acquisition unit is implemented by one or more processors ofthe information processing apparatus; a selection unit that selects atype of a registered image to be used for matching from one of thetwo-dimensional registered images and the three-dimensional registeredimages based on the two-dimensional image, prior to the matching thetwo-dimensional image with the registered image group, wherein theselection unit is implemented by the one or more processors; a displayinformation generation unit that extracts a plurality of candidates fromthe plurality of registrants and generates display information used fordisplaying the extracted candidates in order in accordance with thesimilarity degree based on the similarity degree obtained by matchingthe two-dimensional image of the matching target with a registered imagegroup, wherein the display information generation unit is implemented bythe one or more processors; and a composite image generation unit thatsuperimposes the three-dimensional registered image of a person selectedfrom a plurality of candidates by a user operation on thetwo-dimensional image and generates a composite image, wherein thecomposite image generation unit is implemented by the one or moreprocessors, wherein the composite image generation unit generates thecomposite image by combining a portion of the three-dimensionalregistered image divided into a plurality of parts and a portion of thetwo-dimensional image divided into a plurality of parts.
 2. Theinformation processing apparatus according to claim 1, wherein theselection unit selects the type of the registered image based on theface orientation of the person in the two-dimensional image.
 3. Theinformation processing apparatus according to claim 2, wherein theselection unit selects the two-dimensional registered image when theangle formed by the face orientation of the person and the referencedirection in the two-dimensional image is within a predeterminedthreshold value, and selects the three-dimensional registered image whenthe angle exceeds the threshold value.
 4. The information processingapparatus according to claim 2 further comprising a matching unit thatmatches the two-dimensional image with the registered image group,wherein the matching unit is implemented by the one or more processors,wherein the matching unit adjusts the face orientation of the registrantin the three-dimensional registered image to be consistent with the faceorientation of a person in the two-dimensional image when thethree-dimensional registered image is used for the matching.
 5. Theinformation processing apparatus according to claim 1, wherein theselection unit selects the type of the registered image based on therelationship between the person and the illumination light in thetwo-dimensional image.
 6. The information processing apparatus accordingto claim 1, wherein the selection unit selects the type of theregistered image based on an expression of the person in thetwo-dimensional image.
 7. The information processing apparatus accordingto claim 1, wherein the selection unit selects the type of theregistered image based on an elapsed period from the capturing date ofthe two-dimensional image.
 8. The information processing apparatusaccording to claim 1, wherein the composite image generation unitperforms an editing process for changing the appearance of the person orthe registrant on at least one of the two-dimensional image, thethree-dimensional registered image, and the composite image whengenerating the composite image.
 9. An information processing methodcomprising: acquiring a two-dimensional image of a person, and aregistered image group including two-dimensional registered images andthree-dimensional registered images of a plurality of registrants; andselecting a type of a registered image to be used for matching from oneof the two-dimensional registered images and the three-dimensionalregistered images based on the two-dimensional image, prior to thematching the two-dimensional image with the registered image group;extracting a plurality of candidates from the plurality of registrants,and generating display information used for displaying the extractedcandidates in order in accordance with the similarity degree based onthe similarity degree obtained by matching the two-dimensional image ofthe matching target with a registered image group; and superimposing thethree-dimensional registered image of a person selected from a pluralityof candidates by a user operation on the two-dimensional image, andgenerating a composite image, wherein the composite image is generatedby combining a portion of the three-dimensional registered image dividedinto a plurality of parts and a portion of the two-dimensional imagedivided into a plurality of parts.
 10. An information processingapparatus comprising: an acquisition unit that acquires atwo-dimensional image of a person, and a registered image groupincluding two-dimensional registered images and three-dimensionalregistered images of a plurality of registrants, wherein the acquisitionunit is implemented by one or more processors of the informationprocessing apparatus; a selection unit that selects a type of aregistered image to be used for matching from one of the two-dimensionalregistered images and the three-dimensional registered images based onthe two-dimensional image, prior to the matching the two-dimensionalimage with the registered image group, wherein the selection unit isimplemented by the one or more processors; and a composite imagegeneration unit that superimposes the three-dimensional registered imageof a person selected from a plurality of candidates by a user operationon the two-dimensional image and generates a composite image, whereinthe composite image generation unit is implemented by the one or moreprocessors, wherein the composite image generation unit performs anediting process of at least one of a body hair, an attachment, a bodyshape, a scar, and a change over the years on at least one of thetwo-dimensional image, the three-dimensional registered image, and thecomposite image when generating the composite image.